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Abstract. Here, a Separation Theorem about K-Independent Subspace Analysis (K £ {R, C} real or complex), a 
generalization of EC-Independent Component Analysis (K-ICA) is proven. According to the theorem, K-ISA estimation 
can be executed in two steps under certain conditions. In the first step, 1-dimensional K-ICA estimation is executed. 
In the second step, optimal permutation of the K-ICA elements is searched for. We present sufficient conditions for the 
K-ISA Separation Theorem. Namely, we shall show that (i) spherically symmetric sources (both for real and complex 
cases), as well as (ii) real 2-dimensional sources invariant to 90° rotation, among others, satisfy the conditions of the 
theorem. 

1 Introduction 
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.(Real) Independent Component Analysis (R-ICA) |ll2j aims to recover linearly or non-linearly mixed independent and 
J> hidden sources. There is a broad range of appHcations for R-ICA, such as blind source separation and bhnd source 
.deconvolution feature extraction 0], denoising [5J. Particular applications include, e.g., the analysis of financial data 
'0, data from neurobiology, fMRI, EEG, and MEG (see, e.g., [7ii8i | and references therein). For a recent review on R-ICA 
' see ini. 

Original M-ICA algorithms are 1-dimensional in the sense that all sources are assumed to be independent real valued 
.random variables. However, applications where not all, but only certain groups of the sources are independent may 
'have high relevance in practice. In this case, independent sources can be multi-dimensional. For example, consider the 
.generalization of the cocktail-party problem, where independent groups of people are talking about independent topics, 
or that more than one group of musicians are playing at the party. The separation task requires an extension of R-ICA, 
which can be called (Real) Independent Subspace Analysis (R-ISA) ^XDI, Multi-Dimensional Independent Component 
Analysis (MICA) Group ICA ^^'^ Independent Vector Analysis (IVA) Throughout the paper, we shall use 
.the first abbreviation. An important application for M-ISA is, e.g., the processing of EEG-£MRI data 

Efforts have been made to develop R-ISA algorithms |1 111411511(^117118112) . Related theoretical problems concern 
mostly the estimation of entropy or mutual information. In this context, entropy estimation by Edgeworth expansion 
'^j has been extended to more than 2 dimensions and has been used for clustering and mutual information testing p^ . 
/c-nearest neighbors and geodesic spanning tree methods have been applied in ^2] and pHj for the R-ISA problem. Other 
recent approaches search for independent subspaces via kernel methods [igl and joint block diagonaHzation [12] . 

Beyond the case of real numbers, the search for complex components (Complex Independent Component Analysis, 
C-ICA) assumes more and more practical relevance. Such problems include, beyond others, (i) communication systems, 
(ii) biomedical signal processing, e.g., processing of (f)MRI and EEG data, brain modeUing, (iii) radar appHcations, (iv) 
frequency domain methods (e.g., convolutive models). 

There is a large number of existing C-ICA procedures |2ni21l22l23l2l24l25l2fil27l28l29l3ni3Tl32l3 3 34 35 36; . Maximum 
likelihood principle and complex recurrent neural network are used in jJO] , and [25 , respectively. The APEX algorithm is 
based on Hebbian learning 22J. Complex FastICA algorithm can be found in [2SI- More solutions are based on cumulants: 
e.g., 12, the JADE algorithm [23125] . its higher order variants [2^1, and the EASI algorithm family |2I]. 'Rigid-body' 
learning theory is used in J23|. The SOBI algorithm searches for joint diagonalizer matrix, its refined version, the 
WASOBI method approximates by means of weighted nonhnear least squares. There are complex variants of the 
infomax technique, such as the a split-complex [HI] and the fully-complex infomax [^2] procedures. Minimax Mutual 
Information (SS] and strong-uncorrelating transforms |^MI^^5l3(i] make further promising directions. 



An important observation of previous computer studies |lll.{7) is that general M-ISA solver algorithms are not more 
efficient, in fact, sometimes produce lower quality results than simple M-ICA algorithm superimposed with searches for 
the optimal permutation of the components. This observation led to the present theoretical work and to some computer 
studies that have been pubHshed elsewhere j38l39| . We treat both the real and the complex cases. 

This technical report is constructed as follows: Section |2l introduces complex random variables. In SectionElthe K-ISA 
task is described. Section 01 contains our Separation Theorem for the K-ISA task. Sufficient conditions for the theorem 
are provided in Section Conclusions are drawn in Sectional 

2 Basic Concepts: Matrices, Complex Random Variables 

We introduce the basic concepts for using complex random variables. Excellent review can be found in |4()j . 

B"^ is the transposed of matrix B £ C^^^. Complex conjugation is denoted by a bar it concerns all elements of a 
matrix. The transposed complex conjugate of matrix B is the adjoint matrix B* = B"^. Matrix B e C^^^ is called unitary 
if BB* — II, orthogonal if BB^ — I^, where II is the L-dimensional identity matrix. The sets of L x L dimensional 
unitary and orthogonal matrices are denoted by and , respectively. 

A complex-valued random variable u e (shortly complex random variable) is defined as a random variable of 
the form u = Ufi + iui, where the real and imaginary parts of u, i.e., un and u/ g are real random variables, 
i = -s/^. Expectation value of complex random variables is E[u] = i?[u_Rj + iE[ui], and the variable can be character- 
ized in second order by its covariance matrix cov[u] = E[{u — E[u]){u — E[u])*] and by its pseudo-covariance matrix 
pcov[u] = E[{u — E[u]){u — E[u])'-^]. Complex random variable u is called full, if cow[m] is positive definite. Throughout 
this paper all complex variables are assumed to be full (that is, they are not concentrated in any lower dimensional 
complex subspace). 

3 The K-ISA Model 

First, Section l3Jl introduces the K-ISA task. Section is about the ambiguities of the problem. Section 13731 defines the 
entropy based cost function of the K-ISA task. 

3.1 The K-ISA Equations 

We define the complex (and real) ISA task. The two models are treated together by using the joint notation K e {R, C}. 
Assume that we have M hidden and independent components (random variables) and that only the mixture of them is 
available for observation: 

z(t) = As(t), (1) 

where s{t) — [s^{t); . . . ; s^^(t)] is a vector concatenated of components s'"(t) € K''. For fix to, s™(i) is i.i.d. (independent 
identically distributed) in t, s* is independent from s-' , if i ^ j. The total dimension of the components is D := Md. Thus, 
s{t),z{t) e K^. Matrix A G K^^-" is the so called mixing matrix which, according to our assumptions, is invertible. 

The goal of the K-ISA problem is to estimate the original source s(t) and the unknown mixing matrix A (or its inverse 
W, which is called the separation matrix) by using observations z{t) only. We talk about complex ISA task if K = C 
(C-ISA), and real ISA task if K = R (R-ISA). If d = 1, then complex ICA (C-ICA), and real ICA (R-ICA) tasks are 
obtained. 



3.2 Ambiguities of the K-ISA Model 

Identification of the K-ISA model is ambiguous. However, there are obvious ambiguities of the model: hidden components 
can be determined up to permutation of subspaces and invertible transformation within the subspaces. Further details 
concerning ambiguities can be found here: R-ICA |41j, R-ISA C-ICA 4 1 40 42.43j . C-ISA (see Appendix El- 

Ambiguities within subspaces can be lessened. Namely, given our assumption on the invertibility of matrix A, we can 
assume without any loss of generality that both the sources and the observation are white, that is, 

E[s] = 0,cov[s]=Id, (2) 
E[z] =0,cov[z] =Id. (3) 



Below, we treat real and complex cases separately: 



Real case: It then follows that the mixing matrix A and thus the separation matrix W = A ^ are orthogonal: 

Id = cov[z] = E [zz^] = AE [ss^] A^ = AIzjA^ = AA^. (4) 

The ambiguity of the ISA task is decreased by Eqs. (EJ^®: Now, s™ sources are determined up to permutation and 
orthogonal transformation. 

Complex case: It then follows that the mixing matrix A and thus the separation matrix W = A^^ are unitary: 

Id = cov[z] = E [zz*] = AE [ss*] A* = AI^A* = AA*. (5) 
Thus, components s"* are determined up to permutation and unitary transformation within the subspace. 

3.3 The K-ISA Cost Function 

Now we sketch how to transcribe the K-ISA task into the minimization of sum of multi-dimensional entropies for orthogonal 
matrices (in the real case) and for unitary matrices (in the complex case) . We shall use these formulations of the K-ISA 
task to prove the real and complex versions of the Separation Theorem (Section 0J. 

3.3.1 Real Case The M-ISA task can be viewed as the minimization of mutual information between the estimated 
components: 

/(v) 



/(yi,...,y^) := / /(v)ln 



UfilfrniVm) 



dv (6) 



on the orthogonal group (W e O^), where y = Wz, y = [y^; . . . ; y*^] , / and fm are density functions of y and marginals 
y™, respectively. This cost function / is equivalent to the minimization of the sum of d-dimensional entropies, because 



M 



/(y\...,y^) = ^H(y'")-ff(y) (7) 

rn— 1 
M 

= ^H(y'")-F(Wz) (8) 

m—1 

M 

= ^ if (y") - [H{z) + ln(|det(W)|)]. (9) 



Here, H is Shannon's (multi-dimensional) differential entropy defined with logarithm of base e, |-| denotes absolute 
value, 'det' stands for determinant. In the second equality, the y = Wz relation was exploited, and the 

77(Wz) = iJ(z) -|-ln(|det(W)|) (10) 

rule describing transformation of the differential entropy was used. det(W) = 1 because of the orthogonality of W, so 
ln(|det(W)|) = 0. The H{z) term of the cost is constant in W, therefore the M-ISA task is equivalent to the minimization 
of the cost function 

M 

J(W):=$:i/(y™)^^min^. (11) 

m—1 

3.3.2 Complex Case Similarly, the C-ISA task can be viewed as the minimization of mutual information between the 
estimated components [see Eq. ©], but on the unitary group (W g U^). Here, the Shannon entropy of random variable 
9 u (y™, or y) is the entropy of ipv{u) £ M^^, where 

: 9 Uh^ U(^ ^J'j eM^^. (12) 



That is, H{u) := H[(py{u)]. Here: (8) is the Kronecker product, 3? stands for the real part, 9 for the imaginary part, 
subscript 'v' for vector. One can neglect the last term of the H{y) cost function [see, Eq. Q] during optimization (alike 
in the real case). To see this, consider the mapping 



(fiM ■■ C^""^ 9 M M( 



3fJ(.) 

5R(.) 



e R^^""^^, (13) 



where subscript 'M' indicate matrices. Known properties of mappings tp-u, are as follows |4^: 

det[(^M(M)] = |det(M)|2, (14) 
</7,(Mv) =^m(M)(^,(v). (15) 

In words: itTHl describes transformation of determinant, ifT^ll expresses preservation of operation for matrix- vector multi- 
phcation.^ The following relation holds for the entropy transformation of complex variables: 

Lemma 1 (Transformation of entropy for complex variables). Let u G denote a random variable and let 
Y q£Lxl ^ matrix. Then 

H(Vu) = H{u) + In (|dct(V)|^^ (16) 

Proof. 

H{Vu) = H[ip,{Vu)] = H[^m{V)M^)] = H[M^)] + H\ det[(^M(V)]|) = F[(^„(u)] + ln(| det Vp) (17) 
= i?(u) +ln(|detVn (18) 

The above steps can be justified as follows: 

1. the first equation uses the definition of entropy for complex variables, 

2. then we used property lfT5|l . 

3. transformed the entropy of random variables in M^-^ [see, Eq. ifTTijl ]. 

4. exploited Ijl4|l . and 

5. applied the definition of entropy for complex variables again. 

□ 

Thus H{y) = i7(Wz) +ln(l) = H{z), where unitarity of W is exploited. Further, H{z) is not dependent of matrix W 
and thus term H{y) can be neglected during the course of optimization. We conclude that the C-ISA task can be written 
as the minimization of sum of multi-dimensional entropies. The cost function to be optimized within unitary matrices: 

M 

J(W) = y i/(y'") ^ min . (19) 

m— 1 



4 The R-ISA and C-ISA Separation Theorem 

The main result of this work is that the K-ISA task may be accomplished in two steps under certain conditions. In the 
first step K-ICA is executed. The second step is search for the optimal permutation of the K-ICA components. Section ITTl 
is about the real, whereas Section lOl is about the complex case. 



4.1 The R-ISA Separation Theorem 

We shall rely on entropy inequalities f Section [4.1. 111 . Connection to the R-ICA cost function is derived in Section 14.1.21 
Finally, Section 11.1.31 contains the proof of our theorem. 



^ Note that this connection allows one to reduce the C-ISA task (and thus the C-ICA task) to a R-ISA task directly. According 
to our experiences, however, methods that rely on the C-Separation Theorem that we present here are much more efficient. 



4.1.1 EPI-type Relations (Real Case) First, consider the so called Entropy Power Inequality (EPI) 



L 



,2H{Zt,u^)^J2^2Hiu,)^ (20) 



i=l 



where ui,. . . ,ul G M denote continuous random variables (The name of this inequality is E-EPI, because we shall need 
its complex variant later). This inequality holds for example, for independent continuous variables |44j . 
Let ll'll denote the Euchdean norm. That is, for w S 



L 

2 



E^l (21) 



i=l 

where Wi is the coordinate of vector w. The surface of the L-dimensional unit sphere shah be denoted by S'-^(M): 

S^{R) := {w e : ||w|| = 1}. (22) 
KK-EPI is satisfied [on 5*^(8)] then a further inequality holds: 

Lemma 2. Suppose that continuous random variables ui, . . . ,ul G M satisfy the following inequality 

L 

e2ff(i:ti»,«0 > ^e^^^^-'^'^Vwe S^{R). (23) 
This inequality will be called the R-w-EPI condition. Then 

H > J2^^H{u^),yw e 5^(R). (24) 

\i=l / i=l 

Note 1. R-w-EPI holds, for example, for independent variables Ui, because independence is not affected by multiphcation 
with a constant. 

Proof. Assume that w e S'-^(R). Applying In on condition (l23)l . and using the monotonicity of the In function, we can see 
that the first inequality is valid in the following inequality chain 

2H {y, w^u\ > In (j2 e'""^"*"-') = In (j2 ^'"^^'^ ' > E ' 1^ (e'^^"')) = E ' (25) 

\z=l / \i=l / \i=l ) i=l i=l 

Then, 

1. we used the relation 

H{w,u,)^ H{u,)+lii{\w,\) (26) 
for the entropy of the transformed variable. Hence 

^2H{w,u,) ^ g2H(u.)+21n(|»d) ^ g2H(«.) . g21n(|u.i|) ^ ^2H{ui) . ^2^ (27) 

2. In the second inequality, we exploited the concavity of In. □ 

4.1.2 Connection to the Cost Function of the R-ICA Task Now we shall use Lemma|2lto proceed. The R-ISA 
Separation Theorem will be a corollary of the following claim: 

Proposition 1. Let y = [y^; . . . ;y^^] = y(W) = Ws, where W S , y™ is the estimation of the m*^ component of 
the R-ISA task. Let be the coordinate of the m^^ component. Similarly, let s™ stand for the i*^ coordinate of the 
TO*'' source. Let us assume that the s™ sources satisfy condition l|24ll . Then 

M d M d 



EE^(yr)>EE^(^n- (28) 



Proof. Let us denote the element of matrix W by Wij. Coordinates of y and s will be denoted by yi and Si, 

respectively. Further, let S^, . . . , S*^ denote the indices of the 1**, . . . , M*^ subspaces, i.e., := {1, • ■ • , d}, . . . , S*^ := 
{D — d + 1, . . . , D}. Now, writing the elements of the i*'' row of matrix multiplication y — Ws, we have 



(29) 



and thus, 



Hiy^) 



> lE^M^ 



E ^^J^J 



r + ...+ | E^ 

/eg" 



Y.nAH E 



E ^ 



v/egi 



HE^^ E 

viegi / jeg 




1 



+ 1 E ^ 



/ 

E 



( 



' \(Eiegi 



His,) 



E^^ E 



W, 



/eg- W'l 



(30) 
(31) 

(32) 

(33) 

His,) (34) 
(35) 



The above steps can be justified as follows: 

1. Ij3flll : Eq. Ij29|l was inserted into the argument of H. 

2. 113 New terms were added for Lemma [2 

3. (|32ll : Sources s™ are independent of each other and this independence is preserved upon mixing within the subspaces, 
and we could also use Lemma [21 because W is an orthogonal matrix. 



4. 

5. (E3 

6. ^ 



Nominators were transferred into the Ej terms. 
Variables s™ satisfy condition (|2Hl according to our assumptions. 
We simplified the expression after squaring. 



Using this inequality, summing it for i, exchanging the order of the sums, and making use of the orthogonality of matrix 
W, we have 



D 



D 



E^(y^)^E Ew^'.^(-^. 

»=i Vies' 



E <jHis,) 



D 



E E^i:.p^^ 

jggi \i=i I 
J^His,). 



E (lin^His,) 



(36) 

(37) 

(38) 
□ 



Note 2. The proof holds for components with different dimensions. This is also true for the following theorem. 



4.1.3 Proof of the K-ISA Separation Theorem Having this proposition, now we present our main theorem. 



Theorem 1 (Separation Theorem for K-ISA). Presume that the s™ sources of the 'R-ISA model satisfy condi- 
tion and that the ^-ICA cost function J(W) = X^m^i Yh=i H{yT) has minimum (W e 0°). Then it is sufficient 
to search for the minimum of the M.-ISA task as a permutation of the solution of the M-/Cj4 task. Using the concept of 
separation matrices, it is sufficient to explore forms 

Wr.js^ = PWr_/c^, (39) 

where P (g M-'^^-'^) is a permutation matrix to he determined. 

Proof. M-ICA minimizes the l.h.s. of Eq. lf28ll . that is, it minimizes X]m=i X^iLi H (y™). The set of minima is invariant 
to permutations and to changes of the signs. Also, according to Proposition {s™}, i.e., the coordinates of the s™ 
components of the K-ISA task belong to the set of the minima. □ 



4.2 The C-ISA Separation Theorem 

The proof of the complex case is similar to the proof of the real case. The difference is in the EPI-type relations that we 
apply. Procedure: We define a C-EPI property and then a C-w-EPI relation starting from the vector variant of the M-EPI 
relation. Then the proof relies on analogous steps with the real case, that we detail here for the sake of completeness. 

4.2.1 EPI-type Relations (Complex Case) Let us consider the vector variant of the E-EPI relation. 
Lemma 3 (vector-EPI). For independent (finite covariance random variables) Ui, . . . ,ul G holds I461j that 

L 

^2H{Y.ti ".)/« > e2^("*)/«. (40) 
1=1 

Let us define a similar property for complex random variables: 
Definition 1 (C-EPI). We say that random variables ui, . . . ,ul ^ C satisfy relation C-EPI if 

e^^(i:f^i«0 >^e^("'). (41) 

i=l 

Note 3. This holds for independent random variables ui,...,ul G C, because according to vector-EPI {q = 2) 

L 

g2H(i:f^,«,)/2 > ^g2H(«.)/2, (42) 
i=l 

We need to following lemma: 
Lemma 4. Let us assume that random variables ui, . . . , ul G C satisfy condition 

L 

e^^Cti'"."^ > ^eff('^.«.) yMv=[wi;...;wL]e S^{C) (43) 

i=l 

that we shall call condition C-w-EPI. Here, S'^(C) denotes the L-dimensional complex unit sphere, that is 

5^(C) := |w = K; . . . ; wl] G : ^ |m,|' = i| . (44) 

Then ^ ^ 

H >J2\^^\''Hiu^) Vwg5^(C). (45) 



Proof. Assume that w S S^{^1). Applying In on condition (Q^l, and using the monotonicity of the In function, we can see 
that the first inequahty is vaUd in the following inequality chain 



Then, 

1. we used the relation: 

H{wu) ^ H{u) + \n(\wf] {w,ueC) 
for the entropy of the transformed variable (see Lemma QJ. Hence 

gff(«'.«.) ^ gff(u,)+ln(|«;.|2) ^ gifK) . gln(|t«,p) ^ gff(u,) , |^^|2 

2. In the second inequality, we exploited the concavity of In. 



(47) 

(48) 
□ 



4.2.2 Connection to the Cost Function of the C-ICA Task Now we shall use LemmaQlto proceed. The C-ISA 
Separation Theorem will be a corollary of the following claim: 

Proposition 2. Let y — [y^; . . . ; y^^] = y(W) — Ws, where W £ , y™ is the estimation of the to*'' component of 
the C-ISA task. Let y™ be the i*^ complex coordinate of the m*-^ component. Similarly, let s™ stand for the i*^ coordinate 
of the m*^ source. Let us assume that the s™ sources satisfy condition Ij45|l . Then 



M d 



M d 



(49) 



m— 1 i—\ 



m— 1 i— 1 



Proof. Let us denote the (i, j) element of matrix W by Wij. Coordinates of y and s will be denoted by yt and st, respec- 
tively. Let S^, . . . , S*^ denote the indices belonging to the 1"**, . . . , M*'*. subspaces, that is, := {1, . . . , d}, . . . , S^^ := 
{D — d + 1, . . . , D}. Now, writing the elements of the i"' row of matrix multiplication y = Ws, we have 



and thus, 

H{y. 



E 



2 J ''J 



(50) 



H E ^^^ ^^ 



+ E 



+ 



HEi^^n^ E 



w,. 



(EzeS-lM^uP)' 



E iw^M 

vies*' 



H 



MEi^mME 



iegi 



\2\ 2 



E i^^'iO^fE 



w,. 



2 



1 ''J 



E i^mM E 



(51) 

(52) 

(53) 
(54) 



His,) (55) 



(56) 



The above steps can be justified as follows: 



1. H51II : Eq. Ij5()|l was inserted into the argument of H. 

2. 1I52I1 : New terms were added for Lemma 21 

3. ll53)l : Sources s™ are independent of each other and this independence is preserved upon mixing within the subspaces, 
and we could also use Lemma 31 because W is a unitary matrix. 

4. (Ifiljl : Nominators were transferred into the J2j terms. 

5. : Variables s™ satisfy condition II45|I according to our assumptions. 

6. Il56p : We simplified the expression after squaring. 

Using this inequality, summing it for i, exchanging the order of the sums, and making use of the unitary property of 
matrix W, we have 

Ei?(2/.)>E f E + E (57) 

= E (EivK,,f )i^(^,) + ...+ E (flm^nHis,) m 

D 

= E^(^.)- (59) 

3=1 



Note 4- The proof of the proposition is similar when the dimensions of the subspaces are not constrained to be equal. 
The situation is the same in the next theorem. 



4.2.3 Proof of the C-ISA Separation Theorem Having this proposition, now we present our main theorem. 

Theorem 2 (Separation Theorem for C-ISA). Presume that the s'" sources of the C-ISA model satisfy condi- 
tion (I45II . and that J(W) — X]m=i ^ (2^1")' € ^^); *-e-; C-ICA cost function has minimum. Then it is 
sufficient to search for the minimum of the C-ISA task (Wc.jsa) o,s a permutation of the solution of the C-ICA task 
(Wc-iga)- That it, it is sufficient to search in the form 

'Wc.ISA=PWc.IGA, (60) 

where P (g M'^^'^) is the permutation matrix to be determined. 

Proof. C-ICA minimizes the l.h.s. of Eq. lUflll . that is, it minimizes X]m=i SiLi ^ iuT)- The set of minima is invariant for 
permutations and for multipHcation of the coordinates by numbers with unit absolute value, and according to Proposition|2l 
{s™} (i.e., the coordinates of the C-ISA task) is among the minima. 

We can disregard multipHcations with unit absolute values, because unitary ambiguity within subspaces are present 
in the C-ISA task. □ 



5 Sufficient Conditions of the Separation Theorem 



In the Separation Theorem, we assumed that relations 1I2H1 and are fulfilled for the s™ sources in the real and complex 
cases, respectively. Here, we shall provide sufficient conditions when these inequalities are fulfilled. 



5.1 Real Case 



5.1.1 R-w-EPI According to LemmaEl if the M-w-EPI property [i.e., lf23jl ] holds for sources s™, then inequality lf2ijl 
holds, too. 



5.1.2 Real Spherically Symmetric Sources 

Definition 2 (real spherically symmetric variable). A random variable u G R'* is called real spherically symmetric 
(or shortly S.- spherical), if its density function is not modified by any rotation. Formally, if 

u '^i^"' Ou, VO e (61) 

where ^= denotes equality in distribution. 

A K-spherical random variable has a density function (under mild conditions) and this density function takes constant 
values on concentric spheres around the origin. We shall make use of the following well-known properties of spherically 
symmetric variables |47I48) : 

Lemma 5 (Identical distribution of 1-dimensional projections - Real case). Let v denote a d-dimensional 
variable, which is M.- spherically symmetric. Then the projection of^ onto lines through the origin have identical univariate 
distribution. 

Lemma 6 (Momenta - Real case). The expectation value and the variance of a d-dimensional v 'R- spherically sym- 
metric variable are 

E[w] = 0, (62) 
cou[v] = c{onstant) ■ 1^. (63) 

Now we are ready to claim the following theorem. 

Proposition 3. For spherically symmetric sources s™ (m — 1, . . . , M ) with finite covariance Eq. Ij24|l holds. Further, the 
stronger M.-W-EPI property [Eq. Ij23ll / also holds and with equality between the two sides /Vw e S'^{M.)]. 

Proof. Here, we show that the R-w-EPI property is fulfilled with equality for R-spherical sources. According to Ij62|l - (I63|I . 
spherically symmetric sources s™ have zero expectation values and up to a constant multiplier they also have identity 
covariance matrices: 

S[s"] = 0, (64) 
cov[s"'] = c™ • Id. (65) 

Note that our constraint on the R-ISA task, namely that covariance matrices of the s™ sources should be equal to Id, is 
fulfilled up to constant multipliers. 

Let Pw denote the projection to straight line with direction w e S"'(R), which crosses the origin, i.e., 

d 

Pw : R'' 3 u ^ w^u, e R. (66) 

In particular, if w is chosen as the canonical basis vector e^ (all components are 0, except the i*'' component, which 
is equal to 1), then 

Pe,(u)=u,. (67) 

In this interpretation M-w-EPI [see Eq. Ij23|l ] is concerned with the entropies of the projections of the different sources 
onto straight lines crossing the origin. The l.h.s. projects to w, whereas the r.h.s. projects to the canonical basis vectors. 
Let u denote an arbitrary source, i.e., u :— s™. According to Lemma0 distribution of the spherical u is the same for all 
such projections and thus its entropy is identical. That is. 



E distr distr distr , , r~id/jr^\ /^^\ 

w^u^ ^ ui ^ ... ^ Ud, Vwe5"'(R), (68) 

i=l 

H ^Y,^"^Ui] ^ H (ui) = . . . = H (ud) , Vwe^''(R). (69) 



Thus: 



- l.h.s. of R-w-EPI: e'^^^^'h 

- r.h.s. of M-w-EPI: 



i=l i—1 i—1 

At the first step, we used identity H27II for each of the terms. At the second step, Ij69|l was exploited. Then term e^'"^^ 
was pulled out and we took into account that w G 5''*(R). 

□ 

Note 5. We note that sources of spherically symmetric distribution have already been used in the context of R-ISA 
in pni- In that work, a generative model was assumed. According to the assumption, the distribution of the norms of 
sample projections to the subspaces were independent. This way, the task was restricted to spherically symmetric source 
distributions, which is a special case of the general R-ISA task. 

Note 6. Spherical variables as well as their non-degenerate affine transforms, the so called elliptical variables (which are 
equivalent to spherical ones from the point of view of M-ISA) are thoroughly treated in |47I48| . 



5.1.3 Sources Invariant to 90° Rotation In the previous section, we have seen that random variables with density 
functions invariant to orthogonal transformations (R-spherical variables) satisfy the conditions of the R-ISA Separation 
Theorem. For mixtures of 2-dimensional components {d — 2), invariance to 90° rotation suffices. First, we observe that: 

Note 7. In the R-ISA Separation Theorem, it is suflflcient if some orthogonal transformation of the s'" sources, C'"s™ 
(C™ e C*) satisfy the condition II24|I . In this case, the C^s™ variables are extracted by the permutation search after the 
R-ICA transformation. Because the R-ISA identification has ambiguities up to orthogonal transformation in the respective 
subspaces, this is suitable. In other words, for the R-ISA identification the existence of an Orthonormal Basis (ONB) for 
each u := s™ e R'' components is sufficient, on which the 

h ■.R'^ 3 w H[{w,u)] (71) 

function takes its minimum. [Here, the (w, u) :— J2i=i WiUi random variable is the projection of u to the direction 
w e S"'(R).] In this case, the entropy inequality lf2ijl is met with equality on the elements of the ONB. 

Now we present our result concerning to the d = 2 case. 

Proposition 4. Let us suppose, that the density function f of random variable u = {ui,U2){— s™) e R^ exhibits the 
invariance 

f{ui,U2)^ f{-U2,Ui)^ f{~Ui,-U2)^ fiu2,-ui) (Vu G R^) , (72) 

that is, it is invariant to 90° rotation. If function /i(w) = i/[(w, u)] has minimum on the set {w > 0} n S'^(R), it also 
has minimum on an ONB. ^ Consequently, the R-ISA task can be identified by the use of the R-ISA Separation Theorem. 



Proof. Let 

'0 -1 



R 



1 



(73) 



denote the matrix of 90° ccw rotation. Let w e S'^(R). {w,u) e R is the projection of variable u onto w. The value of 
the density function of the random variable (w, u) in < e R (we move t in direction w) can be calculated by integration 
starting from the point wt, in direction perpendicular to w 

4=.y(w)=(w,u)(i) = / /(wi + z)dz. (74) 

Using the supposed invariance of / and the relation (fzHl we have 

/y(w) = /y(Rw) = /y(R2w) = ./i/(R3w), (75) 

^ Relation w > concerns each coordinates. 



where '=' denotes the equality of functions. Consequently, it is enough to optimize h on the set {w > 0}. Let Wmm be 
the minimum of function h on the set iS'^(M) n {w > 0}. According to Eq. Ij75|l . h takes constant and minimal values in 
the 

{■Wnijn, Rw,„i„, R,^W,„i„, It^Wmin} 

points, {vmm, R-Vmin} is a Suitable ONB in Note[2l □ 
Note 8. A special case of the requirement II72|I is invariance to permutation and sign changes, that is 

/(±?/l,±W2) =/(±M2,±Ul). (76) 

In other words, there exists a function : ^ K, which is symmetric in its variables and 

/(u) = g(K|,KI). (77) 

The domain of Proposition Q includes 

1. the formerly presented R-spherical variables, 

2. or more generally, variables with density function of the form 

f(n)=g{^\u..\P^ ip>0). (78) 

In the literature essentially these variables are called LP(M.)-norm sphericals (forp > 1). Here, we use the LP{R)-norm 
spherical denomination in a slightly extended way, for p > 0. 



5.1.4 Takano's Dependency Criterion We have seen that the M-w-EPI property is sufficient for the R-ISA Sep- 
aration Theorem. In |42J, sufficient condition is provided to satisfy the EPI condition. The condition is based on the 
dependencies of the variables and it concerns the 2-dimensional case. The constraint of d = 2 may be generahzed to 
higher dimensions. We are not aware of such generalizations. 

We note, however, that R-w-EPI requires that K-EPI be satisfied on the surface of the unit sphere. Thus it is satisfactory 
to consider the intersection of the conditions detailed in on surface of the unit sphere. 



5.1.5 Summary of Sufficient Conditions (Real Case) Here, we summarize the presented sufficient conditions 
of the R-ISA Separation Theorem. We have proven, that the requirement described by Eq. (121}! for the s™ sources is 
sufficient for the theorem. This holds if the Ij23|l E-w-EPI condition is fulfilled. The stronger R-w-EPI is valid for 

1. sources satisfying Takano's weak dependency criterion, 

2. R-spherical sources (with equality), 

3. sources invariant to 90° rotation (for d = 2). Specially, (i) variables invariant to permutation and sign changes, and 
(ii)LP(R)-norm spherical variables belong to this family. 

These results are summarized schematically in Table 



5.2 Complex Case 

We provide sufficient conditions that fulfill I^SJ- 



5.2.1 C-w-EPI According to Lemma^I if the C-w-EPI property [i.e., holds for sources s™, then inequality 
holds, too. 



Table 1. Sufficient conditions for the M-ISA Separation Theorem. 



Takano's dependency 
(d = 2) 



invariance to 90° rotation {d = 2) 



specially^^ 



(with ' = ' for a suitable ONB) 



=^ R-w-EPI 



[with ' = ' for all w e S'^{R)] 



invariance to sign and permutation 

specially 
I 

I/*'(K)-norm spherical (p > 0) 
. .t 

generalization for d — 2 

K-spherical symmetry (or elliptical) 



Equation 12411 : sufEcient 
for the R-Separation Theorem 



5.2.2 Complex Spherically Symmetric Sources A complex random variable is complex spherically symmetric, or 
C-spherical, for short, if its density function - which exists under mild conditions - is constant on concentric complex 
spheres. We shall show that as well as the stronger C-w-EPI relations are fulfilled. We need certain definitions 
and some basic features to prove the above statement. Thus, below we shall elaborate on complex sphericals [i5] . 

Definition 3 (C-spherical variable). A random variable v E is called C-spherical, if u = </?«(v) e R^'' R-spherical 
145JI . Equivalent definition for C-sphericals is that they are invariant to unitary transformations. Formally, if 

V Uv, VU e U'^. (79) 

We need two basic properties of C-sphericals j33] to prove the theorem. These are analogous to Lemma^and Lemma^l 

Lemma 7 (Identical distribution of 1-dimensional projections - Complex case). Projections of C-spherical 
variables u E onto any unit vectors in S'^{C) have identical distributions. Formally, forV wi,W2 G S'^{C) 

w*u'^=''w*ue C. (80) 

Lemma 8 (Momenta - Complex case). For C-spherical variable u E C^: 

E[u] =. 0, (81) 
cov[u] = c • Id. (82) 

We claim the following: 

Proposition 5. C-spherical sources s™ g (m = 1, . . . , M ) with finite covariances satisfy condition (I45|l of the C-ISA 
Separation Theorem. Further, they satisfy C-w-EPI (with equality). 

Proof. According to l(8T|l and ll82|l for C-spherical components s™ E C^: E[s"^] — 0, cow[s'"] — c™ • 1^. Note that our 
constraint on the C-ISA task, namely that covariance matrices of the s™ sources should be equal to identity, is fulfilled 
up to constant multipliers. 

Let P-w denote the projection to straight line with direction w E S'^{C), which crosses the origin, i.e., 

d 

Pw : C 9 U w* • u ^ w^u^ E C. (83) 

1=1 



The left and right hand sides of condition Ij45|l correspond to projection onto vector w, and projections onto vectors 
= [0; 0; 1; 0; ...] (1 in the i*^ position and Os otherwise), respectively, w G S'^{C) <^ w € iS''*(C), because conjugation 
preserves length. Given property Il80|l . the distribution and thus the entropy of these projections are equal. That is (let 
u denote an arbitrary source, i.e., u := s™), 

d 

E distr distr distr rid^^\ /r^4\ 

w,u, = ui = ... = Wd, e S^iC), (84) 

i=l 

H (^^w,u}j = H (ui) = . . . = H {ud) , Vwe5''(C). (85) 

Thus: 

- l.h.s. of C-w-EPI: e^("i). 

- r.h.s. of C-w-EPI: ^ ^ 

i=l i—1 i—1 

At the first step, we used identity lUsjl for each of the terms. At the second step, l(85ll was exploited. Then term e^^"^) 
was pulled out and we took into account that w G S'^{C). 

a 

5.2.3 Summary of Sufficient Conditions (Complex Case) Here, we summarize the presented sufficient conditions 
of the C-ISA Separation Theorem. We have proven, that the requirement described by Eq. for the s™ sources 
is sufficient for the theorem. This holds if the ^41M C-w-EPI condition is fulfilled. The stronger C-w-EPI is valid for 
C-spherically symmetric variables. 

These results are summarized schematically in Table [2| 

Table 2. Sufficient conditions for the C-ISA Separation Theorem. 



... ^ [with '=' for all w e s-'jC)] ^ Equation (113: sufficient 

-spherical symmetry C-w-EPI => „ , ^ n rr-i, 

tor the C-beparation Theorem 



6 Conclusions 

In this paper a Separation Theorem, a decomposition principle, was presented for the K-Independent Subspace Analysis 
(K-ISA) problem. If the conditions of the theorem are satisfied then the K-ISA task can be solved in 2 steps. The first step 
is concerned with the search for 1-dimensional independent components. The second step corresponds to a combinatorial 
problem, the search for the optimal permutation. We have shown that spherically symmetric sources (for the real and 
the complex cases, too) satisfy the conditions of the theorem. For the real case and for 2-dimensional sources {d = 2) 
invariance to 90° rotation, or the Takano's dependency criterion is sufficient for the separation. 

These results underline our experiences that the presented 2 step procedure for solving the K-ISA task may produce 
higher quality subspaces than sophisticated search algorithms [171. 

Finally we mention that the possibility of this two step procedure (for the real case) was first noted in . 



7 Appendix 



A Uniqueness of C-ISA 

Here we provide ambiguities of the C-ISA task. The derivation is similar to that of f42], slight modification is used through 
mappings Lpv,fM [see, Eq. (tT2ll and Eq. ((T3ll ]. 

Notations that we need: Let D = dM . Let GZ(i,K) denote the set of invertible matrices in K^^^. Let us decompose 
matrix V e C^^^ into dxd blocks: V = [V*'J] . ^.^^ (V^'^ e C''^''). We say that matrix V is a dx d Mock-permutation 

matrix^ if there is exactly one index j for V i and exactly one i for V j £ {1, . . . , Af}), that V*^ ^ 0, and further, 
this block can be inverted. Matrices B,C € C^^^ are d-equivalent (notation: B C), if B = CL, where L e c^^x-D 
is a d X d block-permutation matrix.^ Stochastic variable 3u = [u^;...;u^^] is called d-independent, if its parts 
u^, . . . ,u^' g are independent. Using L: if u is d-independent, then Lu is that, too. Stochastic variable u g is 
called normal, if every coordinate is normal. Stochastic variable u £ is called normal, if both 3fJ(u) and 5(u) are 
normal. Matrix B £ C^^-° is called d-admissible, if for decomposition B = [B""^]^^^^ (B^'J £ C^""^) all B*'-?' blocks 
are either invertible or indentically 0. (Note: Choosing the coordinates of matrix B from a continuous distribution, the 
matrix is d-admissible with probability 1.). 

Known properties of (^a/, ip^ beyond [ ljl4ll . II15II ] |45j are: 

(y9M(M) nonsingular (singular) <^ M nonsingular (singular), (87) 

<<5i;(vi -f V2) = (y£'t,(vi) -f (^t,(v2). (88) 

To prove our statement, we use the following corollary of the Multivariate Skitovitch-Darmois theorem: 

Lemma 9 (Corollary 3.3 in [42j). Lei wi = J2m=i B™u™ and-W2 = J2m=i C^u™, where u"* are independent random 
variables from M'*, matrices B'",C™ £ W^^"^ are zeros, or they belong to GL{d,R). Then, u™ belonging to B™C™ ^ 
are normal, provided that Wi and W2 are independent. 

Theorems (Ambiguities of C-ISA). Let Gl{D,C) 9 B [B'''\j=i„m (B*'^ £ C^""^) d-admissible and 
s= [s^; . . . ; s^'^] d-independent D = dM -dimensional variable, and none of the variables s™ £ be normal. If Bs 
is again d-independent, then B is d-equivalent to the identity, that is B I^. 

Proof. Indirect. Let us assume that Bs is d-independent, nonetheless B In does not hold. Then there is a column 
index j and there are row indices i\ ^ ii for which B'^ , B'^ '-^ 7^ (and because B d-admissible, thus they are invertible) 
Let us take the parts that correspond to indices ?i, Z2 off from Bs: 

)ne{l,...,J\/}\j 

3 y'' = B^^'^s^' + J2 B'^'^s™ (90) 

me{l,...,_A/}\j 

Applying (pv, and using properties Ij88ll and Ill5|l we have: 

me{l,....M}\j 

M.^''3Mr')^VM{B''^')M^n+ ^m(B^^'™)^„(s") (92) 

me{l,...,J\/}\j 

Taking advantage of l|H3l: invertibility of B^i'^B^^--?' is inherited to ipM{B''^'^),(PM{B'^'^)- Similarly, matrices <pm(B*''") 
{i £ {ii, 12}, rn 7^ j) are either zero or they are invertible, according to their ancestor B*'™, whether it is zero or invertible. 
If s™ £ are independent then variables (^„(s™) £ R^"* are also independent. Thus, as a result of LemmaEI (pv{s-') is 
normal, meaning - by definition - that is also normal: a contradiction. □ 

^ Note: this is an equivalence relation, indeed, because the set of Ls that satisfy the conditions is closed for inversion and for 
multiplication. 

* Reasoning: if for all j there is at most one block (submatrix), which is non-zero, then: (a) for all j there is exactly one block, 
which is non-zero and then B ~d Id, which is a contradiction, or, (b) there is a j index for which submatrix B*'^ has only zeros, 
and then the invertibility of B is not fulfilled. 



Note 9. |4()j has shown an interesting result: for the complex case and for d = 1 (C-ICA task) certain normal sources 
can be separated. This result (and thus Theorem may be extended to d > 1, too, but we are not aware of such 
generalization. 
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